Systems and methods for automated detection of changes in extent of structures using imagery

ABSTRACT

Systems and methods for automated detection of changes in extent of structures using imagery are disclosed, including a non-transitory computer readable medium storing computer executable code that when executed by a processor cause the processor to: align, with an image classifier model, an outline of a structure at a first instance of time to pixels within an image depicting the structure captured at a second instance of time; assess a degree of alignment between the outline and the pixels depicting the structure, so as to classify similarities between the structure depicted within the pixels of the image and the outline using a machine learning model to generate an alignment confidence score; and determine an existence of a change in the structure based upon the alignment confidence score indicating a level of confidence below a predetermined threshold level of confidence that the outline and the pixels within the image are aligned.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application claims priority to and is a continuationof U.S. patent application Ser. No. 16/891,982, entitled “SYSTEMS ANDMETHODS FOR AUTOMATED DETECTION OF CHANGES IN EXTENT OF STRUCTURES USINGIMAGERY,” filed on Jun. 3, 2020, which claims priority to theprovisional patent application identified by U.S. Ser. No. 62/858,656,titled “SYSTEMS FOR DETECTION OF CHANGES IN EXTENT OF STRUCTURES,” filedon Jun. 7, 2019, the entire contents of each of which are herebyexpressly incorporated herein by reference.

BACKGROUND

The detection of changes in the extent of structures, such as buildings,infrastructure, utility towers, roads, bridges, pipelines, and otherobjects, often requires a person to examine at least a first image and asecond image and make a determination as to whether the object isdifferent between the first image and the second image. This can be atime consuming and expensive process. However, current automatedprocesses for the detection of changes of structures from digital imagesin the field of computerized detections of changes in structures alsohave drawbacks.

Digital images can be described as pixelated arrays of electronicsignals. The array may include three dimensions. Such an array mayinclude spatial (x, y or latitude, longitude) and spectral (e.g., red,green, blue) elements. Each pixel in the image captures wavelengths oflight incident on the pixel, limited by the spectral bandpass of thesystem. The wavelengths of light are converted into digital signalsreadable by a computer as float or integer values. How much signalexists per pixel depends, for example, on the lighting conditions (lightreflection or scattering), what is being imaged, and even the imagedobject's chemical properties.

There are at least two methods that are currently used to detect changesin the extent of large structures. A first method is the moretraditional approach of manual comparisons. Here, a first image ismanually compared to a second image by a person. The person has toidentify a structure that is present in both images and trace theboundary of each structure. The trace is then used to compare the extentof the structures to determine whether there was a change in the extent,and, if there is a change in the extent, measure that change. Thisprocess is very time consuming and expensive and relies on humanperception to align the trace and determine whether or not there is achange. A second method is based on algorithmically tracing the samestructure identified in two images and comparing the trace. This methodis computationally expensive as it requires re-extracting an extent forthe structure in each image for each comparison and may still requirehuman involvement. Further, the method is prone to errors in extractingthe extent, and it is time consuming to process. As discussed below,merely analyzing the images using machine learning will not solve theissues identified for either of these methods.

For machine learning (ML) with digital imagery, the goal is to train acomputer system to deconstruct digital images into clusters ofaggregated pixels and statistically identify correlations in theclusters. The correlations are iteratively evaluated and “learned” fromby the computer system, based on a directive to classify a set ofpatterns as a specific thing. For example, the directive could be toclassify the set of patterns to distinguish between a cat and dog,identify all the cars, find the damage on the roof of a building, and soon. The utilization of neural networks in machine learning is known asdeep learning.

Over many imaged objects, regardless of color, orientation, or size ofthe object in the digital image, these specific patterns for the objectare mostly consistent—in effect they describe the fundamental structureof the object of interest. For an example in which the object is a cat,the computer system comes to recognize a cat in an image because thesystem understands the variation in species, color, size, andorientation of cats after seeing many images or instances of cats. Thelearned statistical correlations are then applied to new data to extractthe relevant objects of interest or information.

Convolutional neural networks (CNN) are machine learning models thathave been used to perform this function through the interconnection ofequations that aggregate the pixel digital numbers using specificcombinations of connections of the equations and clustering the pixels,in order to statistically identify objects (or “classes”) in a digitalimage. Exemplary uses of Convolutional Neural Networks are explained,for example, in “ImageNet Classification with Deep Convolutional NeuralNetworks,” by Krizhevsky et al. (Advances in Neural InformationProcessing Systems 25, pages 1097-1105, 2012); and in “FullyConvolutional Networks for Semantic Segmentation,” by Long et al. (IEEEConference on Computer Vision and Pattern Recognition, June 2015).

Generative adversarial networks (GANs) are neural network deep learningarchitectures comprising two neural networks and pitting one against theother. One neural network, called a Generator, generates new datainstances, while another neural network, called a Discriminator,evaluates the new data instances for authenticity, that is, theDiscriminator decides whether each data instance belongs to the trainingdata set or not. The creation of a generative adversarial network isexplained, for example, in “Generative Adversarial Networks,” byGoodfellow, et al. (Departement d'informatique et de rechercheoperationnelle Universite de Montreal, June 2014).

When using computer-based supervised deep learning techniques, such aswith a CNN, for digital images, a user provides a series of examples ofdigital images of the objects of interest to the computer and thecomputer system uses a network of equations to “learn” significantcorrelations for the object of interest via statistical iterations ofpixel clustering, filtering, and convolving.

The artificial intelligence/neural network output is a similar typemodel, but with greater adaptability to both identify context andrespond to changes in imagery parameters. It is typically a binaryoutput, formatted and dictated by the language/format of the networkused, that may then be implemented in a separate workflow and appliedfor predictive classification to the broader area of interest.

In the technological field of remote sensing, digital images may be usedfor mapping geospatial information. Classifying pixels in an image forgeospatial information purposes has been done through varioustechniques. For example, some CNN-based techniques include SemanticSegmentation (also known as pixel-wise classification or individualpixel mapping) using fully convolutional neural networks (FCN) asdescribed in “Fully Convolutional Networks for Semantic Segmentation,”by Long et al., referenced above. In this technique, each pixel in theimage is given a label or classification based on training dataexamples, as discussed in the general overview above. However, thetechnique is computationally intensive, as it requires resources ofcomputational space, time, and money to assess each individual pixel.

A technique that exists outside of the technological field of geospatialmapping is General Image Classification using a convolutional neuralnetwork (CNN), such as that described by Simonyan et al. in the article“Very Deep Convolutional Networks for Large-Scale Image Recognition”(International Conference on Machine Learning, 2015). In General ImageClassification, rather than individual pixels being labeled, an entireimage is given a generalized label. This is typically a much simpleralgorithm than the FCN Semantic Segmentation, and so may require lesscomputation. However, this method provides less information about animage, as it is limited to the image as an aggregated whole as ageneralization rather than identifying particulars, such as whereobjects in the scene are located within the digital image or whereparticular information is located within the digital image.

What is needed is a system to determine a change in extent of astructure depicted in digital images in which the process is not ascomputationally expensive as FCN Semantic Segmentation (pixel-wiseclassification) but is more accurate and provides more information aboutparts of a digital image than General Image Classification. It is tosuch an improved system to determine a change in extent of a structuredepicted in digital images that the present disclosure is directed.

SUMMARY

The problem of determining a change in extent of a structure depicted indigital images is solved with the systems and methods described herein,including automated extent analysis systems and methods for thedetection of changes in extent of one or more structure.

In one aspect of the present disclosure, a non-transitory computerreadable medium may store computer executable code that when executed bya processor cause the processor to: align, with an image classifiermodel, a structure shape of a structure at a first instance of time topixels within an aerial image depicting the structure, the aerial imagecaptured at a second instance of time; assess a degree of alignmentbetween the structure shape and the pixels within the aerial imagedepicting the structure, so as to classify similarities between thestructure depicted within the pixels of the aerial image and thestructure shape using a machine learning model to generate an alignmentconfidence score; and determine an existence of a change in thestructure based upon the alignment confidence score indicating a levelof confidence below a predetermined threshold level of confidence thatthe structure shape and the pixels within the aerial image are aligned.

In one aspect of the present disclosure, the machine learning model maybe a convoluted neural network image classifier model and/or agenerative adversarial network image classifier model.

In one aspect of the present disclosure, the computer executableinstructions when executed by the processor may cause the processor to:identify a shape of the change in the structure using any one or moreof: a point cloud estimate, a convolutional neural network, a generativeadversarial network, and a feature detection technique.

In one aspect of the present disclosure, aligning the structure shapefurther comprises: creating the structure shape using an image of astructure captured at the first instance in time.

In one aspect of the present disclosure, the alignment confidence scoremay be determined by analyzing shape intersection between the structureshape and an outline of the structure depicted within the pixels of theaerial image.

In one aspect of the present disclosure, the structure shape may be apreviously determined outline of the structure at the first instance oftime.

In one aspect of the present disclosure, the first instance of time maybe before the second instance of time or the first instance of time maybe after the second instance of time.

In one aspect of the present disclosure, aligning the structure shapemay further comprise: detecting edges of the structure in the aerialimage; determining one or more shift distance between the structureshape and one or more edges of the detected edges of the structure inthe aerial image; and shifting the structure shape by the shiftdistance.

In one aspect of the present disclosure, the computer executableinstructions when executed by the processor may cause the processor to:determine a structural modification based on the existence of the changeand on a comparison between the structure shape and the pixels withinthe aerial image depicting the structure, after the structure shape isshifted by the shift distance.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate one or more implementationsdescribed herein and, together with the description, explain theseimplementations. The drawings are not intended to be drawn to scale, andcertain features and certain views of the figures may be shownexaggerated, to scale or in schematic in the interest of clarity andconciseness. Not every component may be labeled in every drawing. Likereference numerals in the figures may represent and refer to the same orsimilar element or function. In the drawings:

FIG. 1 is a process flow diagram of an exemplary embodiment of an extentanalysis method in accordance with the present disclosure.

FIG. 2A is a process flow diagram of an exemplary change decision methodin accordance with the present disclosure.

FIG. 2B is a process flow diagram of another exemplary change decisionmethod in accordance with the present disclosure.

FIG. 3 is an exemplary extent analysis system in accordance with thepresent disclosure.

FIG. 4A is an exemplary nadir image depicting a structure of interest ata first instance of time in accordance with the present disclosure.

FIG. 4B is an exemplary image depicting a structure shape of thestructure of FIG. 4A in accordance with the present disclosure.

FIG. 5 is an exemplary nadir image depicting the structure of interestat a second instance of time in accordance with the present disclosure.

FIG. 6 is the image of FIG. 5 depicting the structure at the secondinstance in time overlaid with an unaligned structure shape depicted inFIG. 4B at the first instance in time.

FIG. 7 is the image of FIG. 4 depicting the structure at the secondinstance in time overlaid with an aligned structure shape of FIG. 4Bdepicting the structure shape at the first instance in time and showingan identified change in an extent of the structure.

DETAILED DESCRIPTION

Before explaining at least one embodiment of the disclosure in detail,it is to be understood that the disclosure is not limited in itsapplication to the details of construction, experiments, exemplary data,and/or the arrangement of the components set forth in the followingdescription or illustrated in the drawings unless otherwise noted.

The disclosure is capable of other embodiments or of being practiced orcarried out in various ways. For instance, although extent change of astructure may be used as an example, the methods and systems may be usedto assess other characteristics (by way of example and not limited to,changes in structure footprint or structure area) of other man-madeobjects, non-exclusive examples of which include buildings such asresidential buildings, industrial buildings, or commercial buildings andinclude infrastructure such as roads, bridges, utility lines, pipelines,utility towers. Also, it is to be understood that the phraseology andterminology employed herein is for purposes of description, and shouldnot be regarded as limiting.

As used in the description herein, the terms “comprises,” “comprising,”“includes,” “including,” “has,” “having,” or any other variationsthereof, are intended to cover a non-exclusive inclusion. For example,unless otherwise noted, a process, method, article, or apparatus thatcomprises a list of elements is not necessarily limited to only thoseelements, but may also include other elements not expressly listed orinherent to such process, method, article, or apparatus.

Further, unless expressly stated to the contrary, “or” refers to aninclusive and not to an exclusive “or”. For example, a condition A or Bis satisfied by one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the inventive concept. Thisdescription should be read to include one or more, and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.Further, use of the term “plurality” is meant to convey “more than one”unless expressly stated to the contrary.

As used herein, qualifiers like “substantially,” “about,”“approximately,” and combinations and variations thereof, are intendedto include not only the exact amount or value that they qualify, butalso some slight deviations therefrom, which may be due to computingtolerances, computing error, manufacturing tolerances, measurementerror, wear and tear, stresses exerted on various parts, andcombinations thereof, for example.

As used herein, any reference to “one embodiment,” “an embodiment,”“some embodiments,” “one example,” “for example,” or “an example” meansthat a particular element, feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment and may be used in conjunction with other embodiments. Theappearance of the phrase “in some embodiments” or “one example” invarious places in the specification is not necessarily all referring tothe same embodiment, for example.

The use of ordinal number terminology (i.e., “first”, “second”, “third”,“fourth”, etc.) is solely for the purpose of differentiating between twoor more items and, unless explicitly stated otherwise, is not meant toimply any sequence or order or importance to one item over another orany order of addition.

The use of the term “at least one” or “one or more” will be understoodto include one as well as any quantity more than one. In addition, theuse of the phrase “at least one of X, V, and Z” will be understood toinclude X alone, V alone, and Z alone, as well as any combination of X,V, and Z.

Circuitry, as used herein, may be analog and/or digital components, orone or more suitably programmed processors (e.g., microprocessors) andassociated hardware and software, or hardwired logic. Also, “components”may perform one or more functions. The term “component,” may includehardware, such as a processor (e.g., microprocessor), an applicationspecific integrated circuit (ASIC), field programmable gate array(FPGA), a combination of hardware and software, and/or the like. Theterm “processor” as used herein means a single processor or multipleprocessors working independently or together to collectively perform atask.

Software may include one or more computer readable instructions orcomputer readable code that when executed by one or more componentscause the component to perform a specified function. It should beunderstood that the algorithms described herein may be stored on one ormore non-transitory computer readable medium. Exemplary non-transitorycomputer readable mediums may include random access memory, read onlymemory, flash memory, and/or the like. Such non-transitory computerreadable mediums may be electrically based, magnetically based,optically based, and/or the like.

Referring now to the drawings, FIG. 1 is a process flow chart depictingan exemplary extent analysis method 10. The extent analysis method 10may be implemented by an extent analysis system 100 having one or morecomputer processor 264 (FIG. 3 ). In general, in the extent analysismethod 10, the extent analysis system 100 determines the existence ornon-existence of a change to a structure 64 by attempting to align astructure shape 90 of the structure 64 with a comparative image 110depicting a comparative structure 64′. If the structure shape 90 can bealigned with the comparative structure 64′ (and the alignment is at orabove a predetermined level of confidence) within a predetermined numberof iterations of attempting to align the structure shape 90 with thecomparative structure 64′, then the extent analysis system 100determines that the structure shape 90 and the comparative structure 64′are the same and that there has been no change between the structure 64and the comparative structure 64′. However, if the extent analysissystem 100 cannot confidently align the structure shape 90 with thecomparative structure 64′ within the predetermined number of attempts toalign the structure shape 90 with the comparative structure 64′, thenthe extent analysis system 100 determines that the structure shape 90and the comparative structure 64′ are different, that is, that there hasbeen a change between the structure 64 and the comparative structure64′. In one embodiment, if the extent analysis system 100 determinesthat there has been a change, then the extent analysis system 100implements a change decision method 140 a, 140 b (FIGS. 2A and 2B) todetermine what has changed (for example, structural modifications,additions, demolitions) between the structure 64 and the comparativestructure 64′, such as in the form of a change in extent 130.

More specifically, the extent analysis method 10 utilizes the one ormore computer processor 264 to execute software to cause the one or morecomputer processor 264 to execute the steps of the extent analysismethod 10, which comprises, in step 14, aligning a structure shape 90depicting an extent of a structure 64 (illustrated in FIG. 4B anddiscussed below) to a comparative image 110, the comparative image 110being an image of the comparative structure 64′ (illustrated in FIG. 5and discussed below) and, in step 18, generating an alignment confidencescore. The extent analysis method 10 may further comprise determiningwhether the alignment confidence score is at or above an alignmentthreshold (step 22). If the alignment confidence score is at or abovethe alignment threshold, the extent analysis method 10 makes adetermination that the extent has not changed (step 26). Otherwise, ifthe alignment confidence score is below the alignment threshold, theextent analysis method 10 determines if an iteration count indicative ofa number of iterations of the extent analysis method 10 is at or above apredetermined iteration threshold (step 30). If the number of iterationsis above the iteration threshold in conjunction with the alignmentconfidence score being below the alignment threshold, then the extentanalysis method 10 determines that the extent of the structure 64 haschanged (step 34).

Otherwise, if the alignment confidence score is below the alignmentthreshold and the iteration count is below the iteration threshold, instep 38, the extent analysis system 100 may then apply edge detection tothe comparative image 110, and may extract edges of the comparativestructure 64′ in the comparative image 110. The extent analysis system100 may increase the iteration count by one. The extent analysis method10 may further comprise estimating a shift distance with a pixeldistance buffer using a correlation model to match the structure shape90 to the extracted edges (step 42); and shifting the structure shape 90by the shift distance (step 46), in an attempt to align the structureshape 90 with the comparative structure 64′. The extent analysis method10 may increase the iteration count and then returns to step 18.

The iteration count is a count of a number of loops performed of step 18to step 46 of the extent analysis method 10 when realigning thestructure shape 90 to pixels within the comparative image 110 depictingthe comparative structure 64′.

The extent analysis method 10 may further comprise receiving aparticular location of the structure 64, such as street address orlatitude/longitude, to identify the structure 64 to determine whether ornot the structure 64 has changed. In other embodiments, the extentanalysis method 10 may receive an identification of a geographic area,and then conduct the extent analysis method 10 on structures 64 withinthe geographic area. The geographic area can be defined in a number ofways such as a neighborhood, street, city, town or county. Further, thegeographic area can be defined by a selection of at least threespatially disposed geographic coordinates. In some embodiments, theextent analysis method 10 may translate (for example, by utilizing ageo-coding provider) structure location information (such as a streetaddress) into a set of coordinates, such as longitude-latitudecoordinates, that can be used to query an imagery database 178 or astructure shape database 174 (discussed below) to obtain imagery thatcan be used to determine the structure shape 90 and to obtain thecomparative image 110. Next, the longitude-latitude coordinates of thestructure 64 may be used to query the imagery database 178 or structureshape database 174 in order to retrieve one or more images or one ormore structure shapes 90 of the structure 64 of interest.

In one embodiment, the structure shape 90 used in step 14 may be avector boundary of an outline describing the extent of the structure 64.In one embodiment, the structure shape 90 describes a portion of thestructure 64 that consists only of a building (to the exclusion of agarden, a sidewalk, a driveway, an outdoor kitchen, a pool, etc. thatmay be co-located, adjacent to, or overlapping with the building),whereas, in other embodiments, the structure shape 90 may describe aportion of the structure 64 that includes a building and any adjacentfeatures, such as a porch, driveway, patio, gazebo, pergola, awning,carport, shed, or any other feature that may be adjacent to thebuilding. In some cases, the feature(s) is attached to the building. Forexample, the feature can be an attached porch, awning or carport. In oneembodiment, the structure shape 90 is three dimensional, such as aseries of edges and nodes defining a three-dimensional wireframe outlineof the structure 64, while in other embodiments, the structure shape 90is two dimensional, such as a two-dimensional structure outline.

The alignment confidence score of step 18 is indicative of the degree towhich the structure shape 90 and the structure 64 within the comparativeimage 110 overlay one another, wherein edges of the structure shape 90and pixels within the comparative image 110 depicting the edges of thestructure 64 are substantially collinear. The alignment confidence scoremay be based on an assessment of the degree to which the structure shape90 and the comparative structure 64′ within the comparative image 110overlay one another, that is, to what degree the edges of the structureshape 90 and the pixels depicting the edges of the comparative structure64′ in the comparative image 110 are substantially collinear. Thealignment confidence score may be a probability. In one embodiment, thealignment confidence score may be the probability that the structureshape 90 is aligned with the comparative structure 64′.

In one embodiment, generating the alignment confidence score in step 18is performed by a neural network image classifier model. The neuralnetwork image classifier model may be trained to assess alignment andproduce the alignment confidence score. The neural network imageclassifier model may be any of a generative adversarial model, aconvoluted neural network, a fully convoluted neural network, any otherneural network suitable for generating the alignment confidence score,or any combination of these networks.

The neural network image classifier model may output the alignmentconfidence score as a probability that the edges of the structure shape90 and the edges of the structure 64 in the comparative image 110 aresubstantially collinear. For example, when assessing the alignment ofthe structure shape 90 and the edges of the structure 64, the model maydetermine there is a 95% level of confidence that all sides of thestructure shape 90 align to the extracted edges of the structure 64. Asanother non-exclusive example, the model may determine there is an 85%level of confidence that the structure shape 90 aligns to the edges ofthe structure 64 for each edge of the structure 64.

The alignment threshold is a minimum limit of the alignment confidencescore, above which the extent analysis system 100 determines that thestructure 64 has not changed. In other words, if the alignmentconfidence score is at or above the alignment threshold, it indicatesthat the structure 64 has not changed, because the alignment confidencescore indicates an acceptable (based on the alignment threshold) levelof confidence that the structure shape 90 is aligned with thecomparative structure 64′ in the comparative image 110. If the alignmentconfidence score is under the alignment threshold, it indicates that thestructure 64 has changed, because it indicates that there is a low(based on the alignment threshold) level of confidence that thestructure shape 90 is aligned with the comparative structure 64′ in thecomparative image 110 (that is, the structure shape 90 does not matchthe comparative structure 64′ in the comparative image 110). Thealignment threshold may be set by a user or may be determined by aneural network model trained to make such a determination. For example,the alignment threshold may be set or determined based on the level ofprocessing desired and/or based on the accuracy of the output results.In one embodiment, the alignment threshold may be set or determined tobe 60%, 70%, 80%, 90%, or to be any percentage that is desired by theuser. The alignment threshold may be at or above the set value or simplyabove the set value.

The iteration threshold in step 30 may be a maximum number of iterationsof the loop performed between step 18 and step 46 before a determinationis made that the structure has changed (step 34). The iterationthreshold may be set by a user or may be determined by an artificialintelligence model, e.g., a neural network model trained to make such adetermination. If the neural network does not reach (or exceed) thealignment threshold after a number of iterations equal to the iterationthreshold, the structure is considered to be changed. The iterationthreshold may be set or determined based on the level of processingdesired and/or based on the accuracy of the output results. In oneembodiment, the iteration threshold may be set or determined to be 1,10, 100, 1000, or to be any iteration count that is desired by the user.

In step 38, the extent analysis method 10 may apply edge detectionanalysis to the comparative image 110, and may extracts edge for thecomparative structure 64′ in the comparative image 110. Edge detectionis the process of determining key-points and/or junctions within animage that are consistent with an edge. In one embodiment, edgedetection and extraction may be performed using a combination ofcomputer vision techniques, such as Canny Edge detection and fused withLine-Segment-Detection (LSD), or an artificial intelligence techniqueutilizing convolutional neural networks trained to find key-pointsand/or junctions consistent with edges within an image.

In step 42, the extent analysis method 10 may estimate the shiftdistance with a pixel distance buffer using a correlation model to matchthe structure shape 90 to the extracted edges. The shift distance (whichmay also be known as pixel shift distance) is a distance the structureshape 90 would be adjusted in order to align to an extracted edgedepicted within the comparative image 110. In one embodiment, the shiftdistance may be determined based on computer visionline-segment-detection and a 2D cross correlation method. The shiftdistance may be computed then adjusted with a pixel distance buffer tolimit the shift. In one embodiment, the pixel distance buffer may bedetermined by seeding with a random number or may be determined by anumber of pixels within a certain distance from the structure shape 90.The pixel distance buffer may scale with the resolution of thecomparative image 110 such that a higher-resolution image may have ahigher pixel distance buffer than a lower-resolution image of the samegeographical area wherein the pixel distance buffer of thehigher-resolution image and the pixel distance buffer of thelower-resolution image represent an equivalent real-world distance. Inanother embodiment, a neural network approach is used to determine theshift distance. In the neural network approach, the shift distance maybe based on feature matching performed in the convolutional neuralnetwork.

In one embodiment, the base image 60 and/or the comparative image 110may have a resolution between approximately three inches andapproximately six inches. In one embodiment, the base image 60 and/orthe comparative image 110 may have a resolution of more thanapproximately six inches.

After the shift distance is estimated, in step 46 the extent analysissystem 100 may shift the structure shape 90 a distance equal to theestimated shift distance.

After the extent analysis method 10 determines the structure 64 haschanged in step 34 of the extent analysis method 10, then the changedecision method 140 a, 140 b may be implemented to detect the change inextent 130. The one or more computer processor 264 may execute computersoftware to determine the change in extent 130 of the structure 64 tothe comparative structure 64′, by carrying out the change decisionmethod 140 a, 140 b. Determining the change in extent 130 of thestructure 64 may be accomplished by utilizing a neural net (as shown inFIG. 2A), or first creating the structure shape 90 for the image 60 fromthe first instance in time and then comparing that structure shape 90 topixels depicting the structure 64′ in the comparative image 110 from thesecond instance in time (as shown in FIG. 2B). Additionally, oralternatively, determining the change in extent 130 of the structure 64may be accomplished using other comparative techniques.

In general, to determine the change in extent 130, the extent analysissystem 100 may analyze the structure shapes 90, analyze the comparativeimages 110, and/or analyze the base images 60. A change solution mayinclude a determination of whether there is a change in extent 130between the structure shape 90 and the comparative structure 64′ and maydefine and/or categorize the change, such as by identifying changedareas as compared to the structure shape 90. In one embodiment, theextent analysis system 100 implements a change decision method 140 autilizing a trained machine learning system to detect the change inextent 130. In one embodiment, the extent analysis system 100 may be aspatially aware system and/or the base images 60 and structure shapes 90may be stored in a spatially relational database.

FIG. 2A illustrates one exemplary embodiment of the change decisionmethod 140 a of analyzing the base images 60 and the comparative images110 to identify the change in extent 130 of one or more of thestructures 64. In step 214, the change decision method 140 a uses aneural network to produce a response regarding any change in extent 130of the structure 64 from the base images 60 and the comparative images110. Then, the change decision method 140 a branches to step 218 todetermine whether there is the change in extent 130 based upon theresponse from the neural net.

FIG. 2B depicts another exemplary embodiment of a change decision method140 b for determining the change in extent by analyzing the base image60 from the first instance in time and comparing to the comparativeimage 110 from the second instance in time. The change decision method140 b generally includes creating the structure shape 90 for the baseimage 60 from the first instance in time (step 234), detecting thecomparative structure 64′ in the comparative image 110 from the secondinstance in time (step 238), correlating the structure shape 90 for theimage 60 from the first instance in time with the comparative structure64′ in the comparative image 110 from the second instance in time (step242), and determining, based on the amount of shape intersection betweenthe structure shape 90 and the comparative structure 64′ in thecomparative image 110 from the second instance in time, the change inextent (step 246). In one embodiment, the correlation may be doneutilizing a neural network.

In one embodiment, the one or more computer processor 264 may identify ashape of the change in extent 130 in the comparative structure 64′ usingany one or more of: a point cloud estimate, a convolutional neuralnetwork, a generative adversarial network, and a feature detectiontechnique.

In one embodiment, if the extent analysis system 100 determines thatthere exists the change in extent 130, then the extent analysis system100 may determine the change in extent 130 at the second instance oftime. The extent analysis system 100 may then store the change in extent130 at the second instance of time into one or more database. The extentanalysis system 100 may store one or more of the following in one ormore database: a meta-data of the extent, a timestamp of the creation ofthe comparative image 110, a GPS location of the change in extent 130,the area of the change in extent 130, and other information about thechange in extent 130. By storing this information when it is shown thatthe change in extent 130 exists, the extent analysis system 100 candecrease processing time for future comparisons of the change in extent130 at the second instance of time, and with the structure 64 or thecomparative structure 64′ at a third instance in time.

The extent analysis method 10 and the change decision method 140 a, 140b may be carried out with the extent analysis system 100. Depicted inFIG. 3 is an exemplary embodiment of the extent analysis system 100. Inone embodiment, the extent analysis system 100 may comprise the one ormore computer processor 264 and one or more database.

In one embodiment, the one or more database may comprise an existingstructure shape database 174 and a new imagery database 178. Theexisting structure shape database 174 may store one or more of thestructure shapes 90. The structure shapes 90 may be derived from one ormore base image 60 captured at the first instance in time depicting thestructure 64 at the first instance of time. The new imagery database 178may store one or more of the comparative images 110. The comparativeimage(s) 110 within the new imagery database 178 may be captured at thesecond instance of time and may depict the comparative structure 64′,which is typically the structure 64 at the second instance of time (butmay be the absence of or addition of more structures 64).

The extent analysis system 100 may further comprise one or morenon-transitory memory 268. The computer processor 264 may include (or becommunicatively coupled with) one or more communication component 270.The non-transitory memory 268 may store the existing structure shapesdatabase 174, the new imagery database 178, the existing imagerydatabase 186, the aligned and updated structure shapes database 194, andthe confidence of change database 198, and computer software. Theexisting structure shapes database 174, the new imagery database 178,the existing imagery database 186, the aligned and updated structureshapes database 194, and the confidence of change database 198 may beseparate databases, or may be integrated into a single database. Thecomputer system 260 may include a network 272 enabling bidirectionalcommunication between the computer processor 264 and the non-transitorymemory 268 with a plurality of user devices 284. The user devices 284may communicate via the network 272 and/or may display information on ascreen 296. The computer processor 264 or multiple computer processors264 may or may not necessarily be located in a single physical location.

In one embodiment, the network 272 is the Internet and the user devices284 interface with the computer processor 264 via the communicationcomponent 270 using a series of web pages 288. It should be noted,however, that the network 272 may be almost any type of network and maybe implemented as the World Wide Web (or Internet), a local area network(LAN), a wide area network (WAN), a metropolitan network, a wirelessnetwork, a cellular network, a Global System for Mobile Communications(GSM) network, a code division multiple access (CDMA) network, a 3Gnetwork, a 4G network, a 5G network, a satellite network, a radionetwork, an optical network, a cable network, a public switchedtelephone network, an Ethernet network, combinations thereof, and/or thelike. It is conceivable that in the near future, embodiments of thepresent disclosure may use more advanced networking topologies.

In one embodiment, the computer processor 264 and the non-transitorymemory 268 may be implemented with a server system having multipleservers in a configuration suitable to provide a commercialcomputer-based business system such as a commercial web-site and/or datacenter.

The one or more computer processor 264 may execute computer software toalign the structure shapes 90 to the structures 64 depicted in thecomparative images 110 stored within the new imagery database 178. Inone embodiment, the one or more computer processor 264 may executecomputer software to store the aligned structure shapes 90 in the one ormore database, such as in an updated structure shapes database 194. Inone embodiment, the one or more computer processor 264 may executecomputer software to store the alignment confidence scores in the one ormore database, such as in a confidence of change database 198. Thoughthe databases 174, 178, 186, 194, and 198 are shown as separate entitiesfor clarity, it will be understood that one or more or all of thedatabases 174, 178, 186, 194, and 198 may be combined.

Examples of the extent analysis system 100, extent analysis method 10,and change decision method 140 a, 140 b in use will now be described.Referring now to FIG. 4A, shown therein is an exemplary base image 60captured at a first instance in time. In one embodiment, the base image60 is a nadir, or ortho, view of a property 58, including the structure64 and a sidewalk 68. The structure 64 may include numerous featuressuch as, but not limited to: a slab 72 for utilities, a porch awning 76,a garden 80, and a roof 84. While the base image 60 shows only oneproperty 58 having one structure 64, the base image 60 and/or thecomparative image 110 may show more than one property 58, with eachproperty 58 having one or more than one structure 64.

By way of example, the structure 64 depicted in the base image 60 inFIG. 4A is a house, however, the structure 64 may be a naturallyoccurring structure such as a tree or a man-made structure such as abuilding or shed. In general, the structure 64 may be any type of objectfor which an extent may be defined, that is, any structure 64 that has asize (e.g., length, width, and/or area) and a shape that can be comparedto a size and shape of the structure shape 90. It is understood that thebase image 60 may show more than simply a structure 64 and a sidewalk68, but may include any number of objects in the base image 60 near thestructure 64 and/or on the property 58 such as, but not limited to,vegetation, water features such as rivers, lakes, ponds, and creeks,other human-made structures such as roads, trails, or buildings,animals, and/or the like.

The base image 60 may be captured from an aerial perspective over thestructure 64 and/or from a ground-based perspective. With respect to theaerial perspective, the base image 60 may be a nadir image captured froma directly overhead viewpoint, also referred to as an ortho view ornadir view. A nadir image is typically taken directly below and/orvertically downward from a camera lens positioned above the structure 64(as shown in FIG. 4A). The base image 60 may be an oblique imagecaptured from an overhead aerial oblique view. An aerial oblique viewmay be taken from approximately 10 degrees to approximately 75 degreesfrom a nadir direction. In one embodiment, one or more of the baseimages 60 may be nadir image(s) and one or more of the base images 60may be oblique image(s).

In one embodiment, the extent analysis system 100 may identify anystructure 64 that is not depicted in the base image 60 but that isdepicted in the comparative image 110 as a new structure 64. The extentanalysis system 100 may determine a geo-location (e.g.,latitude/longitude) of the structure 64 in the comparative image 110 andthen analyze one or more base image 60 to determine whether thestructure 64 is depicted within the pixels of the base image(s) 60 atother instances of time.

Referring now to FIG. 4B, shown therein is a vector trace 60′ of thebase image 60. The vector trace 60′ includes at least three components,a structure extent 94, the structure shape 90 encompassing orsurrounding the structure extent 94, and an exterior area of a structuretrace 98 outside of the structure shape 90. The structure extent 94 mayalso be called a boundary extent of the structure 64. The structureextent 94 may be determined by manual selection of vertices (corners,etc.) in an ortho image, by using a point cloud, or by using a neuralnetwork. Techniques for making and using a point cloud, and determininga boundary of a roof within the point cloud are discussed in U.S. PatentPublication No. 201662295336, the entire content of which is herebyincorporated herein by reference. The neural network may be, forexample, a convolutional neural network, a generative adversarialnetwork, or any other neural network configured to determine thestructure extent 94.

FIG. 5 shows an exemplary comparative image 110 captured at a secondinstance in time. The second instance of time can be before or after thefirst instance of time. The comparative image 110 depicts a comparativestructure 64′, the comparative structure 64′ being the structure 64 atthe second instance in time. The second instance of time is differentfrom the first instance of time. Thus, any changes to the structure 64which occurred between the first instance of time and the secondinstance of time may be reflected in the comparative structure 64′ inthe comparative image 110.

The comparative structure 64′ may include many of the same features asstructure 64 including the slab 72 for utilities, the porch awning 76,the garden 80 and the roof 84. However, in the example shown in FIG. 3 ,the comparative structure 64′ further includes a structural modification114. While the comparative image 110 depicts one structural modification114, it is understood that the comparative image 110 may depict morethan one structural modification 114, and it is understood that thestructural modification 114 can be any modification made to thestructure 64, any of its features, or the addition of any features. Itis understood that the comparative image 110 may depict a comparativestructure 64′ wherein the comparative structure 64′ has had featuresremoved, such as a comparative structure 64′ wherein the porch awning 76was removed from structure 64 and is no longer depicted in thecomparative image 110. It is further understood that differences in thecomparative image 110 from the base image 60 are not limited to merelythe features of the comparative structure 64′, but may include theaddition of or removal of one or more structures 64 in the base image60, or that are associated with the structure 64 such as a shed,carport, gazebo, or any other structure that may be constructed.

The base image 60 may be an aerial nadir image and the comparative image110 may also be an aerial nadir image. In certain embodiments, the baseimages 60 and the comparative images 110 may be taken from similarviewpoints, i.e., either nadir or oblique. In some embodiments, the baseimages 60 and 110 are captured from similar compass directions, i.e.,North, South, East or West. Similar compass direction, as used herein,refers to images captured within plus or minus thirty-degree compassdirections of one another. In other embodiments, the base image 60 andthe comparative image 110 can be captured from different viewpoints. Insome embodiments, the base image 60 may be an aerial nadir image, andthe comparative image 110 may be an aerial oblique image taken fromapproximately 10 degrees from the nadir direction.

Exemplary image capture systems that can be used to capture the baseimage 60 and/or the comparative image 110 include those disclosed inU.S. Pat. Nos. 7,424,133, 8,385,672, and U.S. Ser. No. 16/226,320(published as US 2019-0149710 A1), the entire contents of each of whichare hereby incorporated herein by reference.

In one embodiment, each of the base images 60 and/or the comparativeimages 110 may have a unique image identifier such as by use ofmetadata, or otherwise stored in such a way that allows a computersystem 260 to definitively identify each of the base images 60 and/orthe comparative images.

In one embodiment, the base images 60 and/or the comparative images maybe geo-referenced, that is, processed such that pixels in the baseimages 60 and/or the comparative images 110 have a determinedgeo-location, such as x, y, and z coordinates and/or latitude,longitude, and elevation coordinates. See, for example, U.S. Pat. No.7,424,133 that describes techniques for geolocating oblique images andmeasuring within the oblique images. The entire content of U.S. Pat. No.7,424,133 is hereby incorporated herein by reference. Also see forexample WO2018071983, titled “An Image Synthesis System”. Thegeo-location data can be stored as metadata within the images or storedseparately from the images and related to the images using any suitabletechnique, such as unique identifiers. The georeferencing informationassociated with the base images 60 and the comparative images 110 can beused to correlate the structure shape 90 with the comparative structure64′ depicted within the comparative image 110. In other embodiments, thebase images 60 and the comparative images 110 are not geo-referenced.

In one embodiment, the base images 60 and the comparative images 110 areconstrained to a given, occupied parcel. By constraining each of thebase images 60 and the comparative images 110 to a given, occupiedparcel, some assumptions can be made about the existence of a structureand any possible changes. For instance, assumptions regarding anychanges in extent may be made that would limit the changes to changesthat may be performed on an occupied parcel. These assumptions areespecially important when making a determination on a type of change inthe extent, such as an addition of a porch or garage. Such assumptionswould also result in efficiency gains in a drawing process for thestructure shape 90 for change detection.

Shown in FIG. 6 is the comparative image 110 of FIG. 5 in which thestructure shape 90 and the structure extent 94 of the structure 64 aresuperimposed onto the comparative structure 64′. In this example, thestructure shape 90 and the structure extent 94 of the structure 64 areunaligned with the comparative structure 64′. In addition to thestructure features of the roof 84, structural modification 114, and slab72 for utilities, FIG. 6 shows a shift difference 120 between thestructure shape 90 and the comparative structure 64′. The shiftdifference 120 may be measured in more than one direction, that is, theshift difference 120 may be measured by a number of pixels in a verticaldirection, the vertical direction being, for example, a direction fromthe lower edge of the image to the upper edge of the image, and/or in ahorizontal direction, the horizontal direction being, for example, thedirection from the left-most edge of the image to the right-most edge ofthe image. The shift difference 120 may be a rotational value. In otherembodiments, the comparative image 110 may be scaled. Scaling thecomparative image 110 may be based, at least in part, on the shiftdistance and the alignment confidence score. The shift difference(s) 120may be measured or determined to produce the shift distance(s).

FIG. 7 shows the comparative image 110 of FIG. 5 and the structure shape90 and the structure extent 94 of structure 64 superimposed and nowaligned onto the comparative structure 64′ in the comparative image 110(such as based on the shift distance). As shown in FIG. 7 , thestructure shape 90 and the structure extent 94 cover the portion of thecomparative structure 64′ that was present in the structure 64 at thefirst instance in time. A portion of the comparative structure 64′ thatis not covered by the structure shape 90 and the structure extent 94 isidentified by a striped region defining a portion of the comparativestructure 64′ that has been added (a structural modification 114) thatmakes up the change in extent 130. The change in extent 130 is thedifference in the extent 94 of the structure 64 at the first instance intime and the extent of the comparative structure 64′ at the secondinstance in time.

The one or more computer processor 264 may identify a shape of thechange in extent 130 in the comparative structure 64′ using any one ormore of: a point cloud estimate, a convolutional neural network, agenerative adversarial network, and a feature detection technique.

From the above description and examples, it is clear that the inventiveconcepts disclosed and claimed herein are well adapted to attain theadvantages mentioned herein. While exemplary embodiments of theinventive concepts have been described for purposes of this disclosure,it will be understood that numerous changes may be made which willreadily suggest themselves to those skilled in the art and which areaccomplished within the spirit of the inventive concepts disclosed andclaimed herein. For exemplary purposes, examples of images 60 and 110 ofresidential structures have been used. However, it is to be understoodthat the example is for illustrative purposes only and is not to beconstrued as limiting the scope of the invention.

The extent analysis system 100, the extent analysis method 10, and thechange decision methods 140 a, 140 b may be used for a wide variety ofreal-world applications with respect to the structure 64. Non-exclusiveexamples of such applications include use of the results of the methodsto determine a tax assessment, provide and/or complete inspections, toevaluate condition, to repair, to create under-writing, to insure, topurchase, to construct, or to value the structure 64. For example, amunicipality may tax real estate property based on the size and type ofthe structures 64 located on the property. Detecting and determining theextent of change to the structures 64 can be used to adjust such taxes.As another example, municipalities may require building permits forchanges to structures 64 or the addition or demolition of structures.Detecting and determining the extent of change to the structures 64 canbe used to monitor such changes, additions, and demolitions. As yetanother example, insurance companies may underwrite and/or pay forrepair of structures 64 based at least in part on size and condition ofthe structures 64. Detecting and determining the extent of change to thestructures 64 can be used to create and/or monitor insuranceunderwriting or assessment.

What is claimed is:
 1. A non-transitory computer readable medium storingcomputer executable code that when executed by a processor cause theprocessor to: align, with an image classifier model, an outline of astructure at a first instance of time to pixels within an imagedepicting the structure, the image captured at a second instance oftime; assess a degree of alignment between the outline and the pixelswithin the image depicting the structure, so as to classify similaritiesbetween the structure depicted within the pixels of the image and theoutline using a machine learning model to generate an alignmentconfidence score; and determine an existence of a change in thestructure based upon the alignment confidence score indicating a levelof confidence below a predetermined threshold level of confidence thatthe outline and the pixels within the image are aligned.
 2. Thenon-transitory computer readable medium of claim 1, wherein the machinelearning model is a convoluted neural network image classifier model. 3.The non-transitory computer readable medium of claim 1, wherein themachine learning model is a generative adversarial network imageclassifier model.
 4. The non-transitory computer readable medium ofclaim 1, further comprising computer executable instructions that whenexecuted by the processor cause the processor to: identify a shape ofthe change in the structure using any one or more of: a point cloudestimate, a convolutional neural network, a generative adversarialnetwork, and a feature detection technique.
 5. The non-transitorycomputer readable medium of claim 1, wherein aligning the outlinefurther comprises: creating the outline using an image of the structurecaptured at the first instance of time.
 6. The non-transitory computerreadable medium of claim 1, wherein the outline is a first outline andwherein the alignment confidence score is determined by analyzing shapeintersection between the first outline and a second outline of thestructure depicted within the pixels of the image.
 7. The non-transitorycomputer readable medium of claim 1, wherein the outline is a vectorboundary describing an extent of the structure at the first instance oftime.
 8. The non-transitory computer readable medium of claim 1, whereinthe first instance of time is before the second instance of time.
 9. Thenon-transitory computer readable medium of claim 1, wherein the firstinstance of time is after the second instance of time.
 10. Thenon-transitory computer readable medium of claim 1, wherein aligning theoutline further comprises: detecting edges of the structure in theimage; determining one or more shift distance between the outline andone or more edges of the detected edges of the structure in the image;and shifting the outline by the shift distance.
 11. The non-transitorycomputer readable medium of claim 10, further comprising computerexecutable instructions that when executed by the processor cause theprocessor to: determine a structural modification based on the existenceof the change and on a comparison between the outline and the pixelswithin the image depicting the structure, after the outline is shiftedby the shift distance.
 12. A method, comprising: aligning, automaticallywith one or more processors utilizing an image classifier model, anoutline of a structure at a first instance of time to pixels within animage depicting the structure, the image captured at a second instanceof time; assessing, automatically with the one or more processors, adegree of alignment between the outline and the pixels within the imagedepicting the structure, so as to classify similarities between thestructure depicted within the pixels of the image and the outline usinga machine learning model to generate an alignment confidence score; anddetermining, automatically with the one or more processors, an existenceof a change in the structure based upon the alignment confidence scoreindicating a level of confidence below a predetermined threshold levelof confidence that the outline and the pixels within the image arealigned.
 13. The method of claim 12, wherein the machine learning modelis at least one of a convoluted neural network image classifier modeland a generative adversarial network image classifier model.
 14. Themethod of claim 12, further comprising: identifying, automatically withthe one or more processors, a shape of the change in the structure usingany one or more of: a point cloud estimate, a convolutional neuralnetwork, a generative adversarial network, and a feature detectiontechnique.
 15. The method of claim 12, wherein aligning the outlinefurther comprises: creating, automatically with the one or moreprocessors, the outline using an image of the structure captured at thefirst instance of time.
 16. The method of claim 12, wherein the outlineis a first outline, and wherein the alignment confidence score isdetermined by analyzing shape intersection between the first outline anda second outline of the structure depicted within the pixels of theimage.
 17. The method of claim 12, wherein the outline is a vectorboundary describing an extend of the structure at the first instance oftime.
 18. The method of claim 12, wherein the first instance of time isbefore the second instance of time or the first instance of time isafter the second instance of time.
 19. The method of claim 12, whereinaligning the outline further comprises: detecting edges of the structurein the image; determining one or more shift distance between the outlineand one or more edges of the detected edges of the structure in theimage; and shifting the outline by the shift distance.
 20. The method ofclaim 19, further comprising: determining, automatically with the one ormore processors, a structural modification based on the existence of thechange and on a comparison between the outline and the pixels within theimage depicting the structure, after the outline is shifted by the shiftdistance.